The present system relates to implementing Free-viewpoint Television (FTV) with minor revision to typical television (TV) infrastructure and related viewer experiences. In addition to home entertainment, FTV can be used in other environments, such as gaming and education.
When viewing typical TV, the viewpoint is predetermined during the production of the particular program being watched by the placement of an acquisition camera. Unlike typical TV, FTV provides the viewer the freedom of choosing his own viewpoint by supplying the viewer's television set, or other display device, with multiple video streams captured by a set of cameras, each depicting a different view of a single scene, and by using the provided ‘real’ views to create a continuum of ‘virtual’ views, as described in detail in the '327 application.
However, the multiple video streams of a single scene may not contain explicit information about their spatial positioning relative to one another, so it is desirable for FTV to determine the relative spatial positioning of the video streams in order to select which video streams to use as the basis in creating the virtual view. Thus, the spatial positioning relationships are extrapolated from the data that is contained in the video streams. A variety of virtual view specification techniques utilize existing, image-based rendering techniques. For example, in a fully calibrated system where the relative spatial relationships between all the video streams are known the virtual view determination may be done through geometry. An alternative technique uses a viewer's manual selection of a variety of points, including the projection of the virtual camera center. Another technique determines the virtual viewpoint using a rotation matrix and a translation vector with respect to a known camera. However, these approaches require a fully calibrated system with known camera positioning or calibration input from a viewer.
The '327 application describes a technique for specifying a virtual view between any two viewer-chosen real basis views in an uncalibrated system. This allows virtual viewpoint specification using only a single user specified parameter, thereby permitting a virtual view to be defined by indicating a one dimensional directional shift, e.g. to the left, to the right, up, or down, from a current view. One limitation of the above two view based specification is the requirement for the viewer to choose the two basis views. This selection requirement may be non-intuitive for the viewer and tends to disrupt the TV viewing experience. For example, every time the viewer begins viewing a new scene it may be necessary to first display the available basis views for that scene and prompt the viewer to select the initial basis views. Thus a more ‘user friendly’ technique for selecting the basis views is desirable.